{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 直接调用XGBoost\n",
    "该示例中我们在xgboost安装包中自带的Mushroom数据集演示直接调用XGBoost\n",
    "1. 读取数据--> DMatrix\n",
    "2. 设置参数\n",
    "3. 模型训练：train/cv\n",
    "    3.1: train with 在校验集上early stop\n",
    "    3.2: cv\n",
    "4. 预测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 导入必要的工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 导入必要的工具包\n",
    "import xgboost as xgb\n",
    "\n",
    "# 计算分类正确率\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据读取"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XGBoost可以加载libsvm格式的文本数据，libsvm的文件格式（稀疏特征）如下：\n",
    "1 101:1.2 102:0.03\n",
    "0 1:2.1 10001:300 10002:400\n",
    "...\n",
    "\n",
    "每一行表示一个样本，第一行的开头的“1”是样本的标签。“101”和“102”为特征索引，'1.2'和'0.03' 为特征的值。\n",
    "在两类分类中，用“1”表示正样本，用“0” 表示负样本。也支持[0,1]表示概率用来做标签，表示为正样本的概率。\n",
    "\n",
    "下面的示例数据需要我们通过一些蘑菇的若干属性判断这个品种是否有毒。\n",
    "UCI数据描述：http://archive.ics.uci.edu/ml/machine-learning-databases/mushroom/ ，\n",
    "每个样本描述了蘑菇的22个属性，比如形状、气味等等（将22维原始特征用加工后变成了126维特征，\n",
    "并存为libsvm格式)，然后给出了这个蘑菇是否可食用。其中6513个样本做训练，1611个样本做测试。\n",
    "\n",
    "XGBoost加载的数据存储在对象DMatrix中\n",
    "XGBoost自定义了一个数据矩阵类DMatrix，优化了存储和运算速度\n",
    "DMatrix文档：http://xgboost.readthedocs.io/en/latest/python/python_api.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# read in data，数据在xgboost安装的路径下的demo目录,现在我们将其copy到当前代码下的data目录\n",
    "dpath = './data/'\n",
    "dtrain = xgb.DMatrix(dpath + 'agaricus.txt.train')\n",
    "dtest = xgb.DMatrix(dpath + 'agaricus.txt.test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看数据情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "127L"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtrain.num_col()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6513L"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtrain.num_row()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1611L"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtest.num_row()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练参数设置"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "max_depth： 树的最大深度。缺省值为6，取值范围为：[1,∞]\n",
    "eta：学习率。为了防止过拟合，更新过程中用到的收缩步长。\n",
    "eta通过缩减特征的权重使提升计算过程更加保守。缺省值为0.3，取值范围为：[0,1]\n",
    "silent：取0时表示打印出运行时信息，取1时表示以缄默方式运行，不打印运行时信息。缺省值为0\n",
    "objective： 定义学习任务及相应的学习目标，“binary:logistic” 表示二分类的逻辑回归问题，输出为概率。\n",
    "\n",
    "其他参数取默认值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# specify parameters via map\n",
    "param = {'max_depth':2, 'eta':1, 'silent':0, 'objective':'binary:logistic' }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有了参数列表和数据就可以训练模型了 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 设置boosting迭代计算次数，即若学习器（决策树）的数目\n",
    "num_round = 2\n",
    "\n",
    "#import time\n",
    "#starttime = time.clock()\n",
    "\n",
    "bst = xgb.train(param, dtrain, num_round)\n",
    "\n",
    "#endtime = time.clock()\n",
    "#print (endtime - starttime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看模型在训练集上的分类性能"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XGBoost预测的输出是概率。这里蘑菇分类是一个二类分类问题，输出值是样本为第一类的概率。\n",
    "我们需要将概率值转换为0或1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Accuary: 97.77%\n"
     ]
    }
   ],
   "source": [
    "train_preds = bst.predict(dtrain)\n",
    "train_predictions = [round(value) for value in train_preds]\n",
    "y_train = dtrain.get_label()\n",
    "train_accuracy = accuracy_score(y_train, train_predictions)\n",
    "print (\"Train Accuary: %.2f%%\" % (train_accuracy * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型训练好后，可以用训练好的模型对测试数据进行预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# make prediction\n",
    "preds = bst.predict(dtest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "检查模型在测试集上的正确率\n",
    "XGBoost预测的输出是概率，输出值是样本为第一类的概率。我们需要将概率值转换为0或1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predictions = [round(value) for value in preds]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 97.83%\n"
     ]
    }
   ],
   "source": [
    "y_test = dtest.get_label()\n",
    "test_accuracy = accuracy_score(y_test, predictions)\n",
    "print(\"Test Accuracy: %.2f%%\" % (test_accuracy * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型可视化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "调用XGBoost工具包中的plot_tree，在显示\n",
    "要可视化模型需要安装graphviz软件包\n",
    "ip install graphviz\n",
    "\n",
    "plot_tree（）的三个参数：\n",
    "1. 模型\n",
    "2. 树的索引，从0开始\n",
    "3. 显示方向，缺省为竖直，‘LR'是水平方向"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\n",
       " -->\n",
       "<!-- Title: %3 Pages: 1 -->\n",
       "<svg width=\"301pt\" height=\"441pt\"\n",
       " viewBox=\"0.00 0.00 301.21 441.13\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 437.13)\">\n",
       "<title>%3</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-437.13 297.206,-437.13 297.206,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"199.688\" cy=\"-358.847\" rx=\"74.0654\" ry=\"74.0654\"/>\n",
       "<text text-anchor=\"middle\" x=\"199.688\" y=\"-354.647\" font-family=\"Times,serif\" font-size=\"14.00\">f60&lt;&#45;9.53674e&#45;07</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"105.688\" cy=\"-160.282\" rx=\"74.0654\" ry=\"74.0654\"/>\n",
       "<text text-anchor=\"middle\" x=\"105.688\" y=\"-156.082\" font-family=\"Times,serif\" font-size=\"14.00\">f29&lt;&#45;9.53674e&#45;07</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"#0000ff\" d=\"M162.555,-294.05C157.632,-284.92 152.795,-275.571 148.46,-266.565 144.224,-257.765 140.027,-248.433 136.012,-239.102\"/>\n",
       "<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"139.19,-237.631 132.064,-229.791 132.746,-240.364 139.19,-237.631\"/>\n",
       "<text text-anchor=\"middle\" x=\"183.301\" y=\"-255.365\" font-family=\"Times,serif\" font-size=\"14.00\">yes, missing</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"293.225,-178.282 198.15,-178.282 198.15,-142.282 293.225,-142.282 293.225,-178.282\"/>\n",
       "<text text-anchor=\"middle\" x=\"245.688\" y=\"-156.082\" font-family=\"Times,serif\" font-size=\"14.00\">leaf=&#45;6.23624</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>0&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"#ff0000\" d=\"M216.425,-286.327C224.382,-252.325 233.421,-213.698 239.321,-188.486\"/>\n",
       "<polygon fill=\"#ff0000\" stroke=\"#ff0000\" points=\"242.775,-189.09 241.645,-178.555 235.959,-187.495 242.775,-189.09\"/>\n",
       "<text text-anchor=\"middle\" x=\"231.688\" y=\"-255.365\" font-family=\"Times,serif\" font-size=\"14.00\">no</text>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\"><title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"97.5632,-36 -0.188225,-36 -0.188225,-0 97.5632,-0 97.5632,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"48.6875\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\">leaf=0.784718</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"#0000ff\" d=\"M78.0224,-91.1959C71.4222,-74.9523 64.7623,-58.5617 59.4602,-45.5126\"/>\n",
       "<polygon fill=\"#0000ff\" stroke=\"#0000ff\" points=\"62.7007,-44.19 55.6938,-36.2431 56.2156,-46.8251 62.7007,-44.19\"/>\n",
       "<text text-anchor=\"middle\" x=\"102.301\" y=\"-56.8\" font-family=\"Times,serif\" font-size=\"14.00\">yes, missing</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\"><title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"210.225,-36 115.15,-36 115.15,-0 210.225,-0 210.225,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"162.688\" y=\"-13.8\" font-family=\"Times,serif\" font-size=\"14.00\">leaf=&#45;0.96853</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>1&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"#ff0000\" d=\"M133.353,-91.1959C139.953,-74.9523 146.613,-58.5617 151.915,-45.5126\"/>\n",
       "<polygon fill=\"#ff0000\" stroke=\"#ff0000\" points=\"155.159,-46.8251 155.681,-36.2431 148.674,-44.19 155.159,-46.8251\"/>\n",
       "<text text-anchor=\"middle\" x=\"154.688\" y=\"-56.8\" font-family=\"Times,serif\" font-size=\"14.00\">no</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x1146989d0>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from matplotlib import pyplot\n",
    "import graphviz\n",
    "#xgb.plot_tree(bst, num_trees=0, rankdir= 'LR' )\n",
    "#pyplot.show()\n",
    "\n",
    "#xgb.plot_tree(bst,num_trees=1, rankdir= 'LR' )\n",
    "#pyplot.show()\n",
    "#xgb.to_graphviz(bst,num_trees=0)\n",
    "xgb.to_graphviz(bst,num_trees=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 采用交叉验证得到最佳的n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_estimators = 1000\n",
    "early_stopping_rounds = 10\n",
    "cv_result = xgb.cv(param, dtrain, num_boost_round=n_estimators, folds =5,\n",
    "             metrics='logloss', early_stopping_rounds=early_stopping_rounds)\n",
    "  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>test-logloss-mean</th>\n",
       "      <th>test-logloss-std</th>\n",
       "      <th>train-logloss-mean</th>\n",
       "      <th>train-logloss-std</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.233672</td>\n",
       "      <td>0.004222</td>\n",
       "      <td>0.233574</td>\n",
       "      <td>0.002588</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.137153</td>\n",
       "      <td>0.004188</td>\n",
       "      <td>0.137062</td>\n",
       "      <td>0.001687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.083247</td>\n",
       "      <td>0.001000</td>\n",
       "      <td>0.083167</td>\n",
       "      <td>0.000856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.057180</td>\n",
       "      <td>0.001501</td>\n",
       "      <td>0.057117</td>\n",
       "      <td>0.001012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.042534</td>\n",
       "      <td>0.000609</td>\n",
       "      <td>0.041473</td>\n",
       "      <td>0.000507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.028232</td>\n",
       "      <td>0.004339</td>\n",
       "      <td>0.028354</td>\n",
       "      <td>0.001794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.019509</td>\n",
       "      <td>0.001554</td>\n",
       "      <td>0.019259</td>\n",
       "      <td>0.000156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0.015521</td>\n",
       "      <td>0.001268</td>\n",
       "      <td>0.014934</td>\n",
       "      <td>0.000269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.011678</td>\n",
       "      <td>0.000404</td>\n",
       "      <td>0.011424</td>\n",
       "      <td>0.000466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.009882</td>\n",
       "      <td>0.000648</td>\n",
       "      <td>0.009627</td>\n",
       "      <td>0.000080</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.007019</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.006814</td>\n",
       "      <td>0.000208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.005447</td>\n",
       "      <td>0.000269</td>\n",
       "      <td>0.005408</td>\n",
       "      <td>0.000180</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0.004590</td>\n",
       "      <td>0.000211</td>\n",
       "      <td>0.004596</td>\n",
       "      <td>0.000129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.003703</td>\n",
       "      <td>0.000252</td>\n",
       "      <td>0.003607</td>\n",
       "      <td>0.000310</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.003236</td>\n",
       "      <td>0.000143</td>\n",
       "      <td>0.003208</td>\n",
       "      <td>0.000254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.002857</td>\n",
       "      <td>0.000251</td>\n",
       "      <td>0.002785</td>\n",
       "      <td>0.000308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.002570</td>\n",
       "      <td>0.000209</td>\n",
       "      <td>0.002509</td>\n",
       "      <td>0.000295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.002249</td>\n",
       "      <td>0.000184</td>\n",
       "      <td>0.002169</td>\n",
       "      <td>0.000217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.002007</td>\n",
       "      <td>0.000134</td>\n",
       "      <td>0.001925</td>\n",
       "      <td>0.000161</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.001768</td>\n",
       "      <td>0.000097</td>\n",
       "      <td>0.001684</td>\n",
       "      <td>0.000111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.001609</td>\n",
       "      <td>0.000077</td>\n",
       "      <td>0.001554</td>\n",
       "      <td>0.000099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.001459</td>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.001408</td>\n",
       "      <td>0.000042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.001347</td>\n",
       "      <td>0.000001</td>\n",
       "      <td>0.001297</td>\n",
       "      <td>0.000016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.001238</td>\n",
       "      <td>0.000022</td>\n",
       "      <td>0.001202</td>\n",
       "      <td>0.000022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0.001155</td>\n",
       "      <td>0.000032</td>\n",
       "      <td>0.001120</td>\n",
       "      <td>0.000018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.001086</td>\n",
       "      <td>0.000031</td>\n",
       "      <td>0.001057</td>\n",
       "      <td>0.000017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.001028</td>\n",
       "      <td>0.000025</td>\n",
       "      <td>0.000990</td>\n",
       "      <td>0.000017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.000991</td>\n",
       "      <td>0.000030</td>\n",
       "      <td>0.000949</td>\n",
       "      <td>0.000018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>0.000947</td>\n",
       "      <td>0.000029</td>\n",
       "      <td>0.000912</td>\n",
       "      <td>0.000024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.000913</td>\n",
       "      <td>0.000031</td>\n",
       "      <td>0.000872</td>\n",
       "      <td>0.000023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>0.000683</td>\n",
       "      <td>0.000018</td>\n",
       "      <td>0.000657</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>119</th>\n",
       "      <td>0.000682</td>\n",
       "      <td>0.000018</td>\n",
       "      <td>0.000657</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>120</th>\n",
       "      <td>0.000682</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000657</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121</th>\n",
       "      <td>0.000681</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000657</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122</th>\n",
       "      <td>0.000681</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000656</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>0.000681</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000656</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>0.000681</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000656</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>125</th>\n",
       "      <td>0.000681</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000656</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>126</th>\n",
       "      <td>0.000681</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000656</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>0.000681</td>\n",
       "      <td>0.000018</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>128</th>\n",
       "      <td>0.000681</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>134</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>137</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000655</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>138</th>\n",
       "      <td>0.000680</td>\n",
       "      <td>0.000019</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>139</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>140</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>141</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>0.000679</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000654</td>\n",
       "      <td>0.000014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>148 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     test-logloss-mean  test-logloss-std  train-logloss-mean  \\\n",
       "0             0.233672          0.004222            0.233574   \n",
       "1             0.137153          0.004188            0.137062   \n",
       "2             0.083247          0.001000            0.083167   \n",
       "3             0.057180          0.001501            0.057117   \n",
       "4             0.042534          0.000609            0.041473   \n",
       "5             0.028232          0.004339            0.028354   \n",
       "6             0.019509          0.001554            0.019259   \n",
       "7             0.015521          0.001268            0.014934   \n",
       "8             0.011678          0.000404            0.011424   \n",
       "9             0.009882          0.000648            0.009627   \n",
       "10            0.007019          0.000654            0.006814   \n",
       "11            0.005447          0.000269            0.005408   \n",
       "12            0.004590          0.000211            0.004596   \n",
       "13            0.003703          0.000252            0.003607   \n",
       "14            0.003236          0.000143            0.003208   \n",
       "15            0.002857          0.000251            0.002785   \n",
       "16            0.002570          0.000209            0.002509   \n",
       "17            0.002249          0.000184            0.002169   \n",
       "18            0.002007          0.000134            0.001925   \n",
       "19            0.001768          0.000097            0.001684   \n",
       "20            0.001609          0.000077            0.001554   \n",
       "21            0.001459          0.000023            0.001408   \n",
       "22            0.001347          0.000001            0.001297   \n",
       "23            0.001238          0.000022            0.001202   \n",
       "24            0.001155          0.000032            0.001120   \n",
       "25            0.001086          0.000031            0.001057   \n",
       "26            0.001028          0.000025            0.000990   \n",
       "27            0.000991          0.000030            0.000949   \n",
       "28            0.000947          0.000029            0.000912   \n",
       "29            0.000913          0.000031            0.000872   \n",
       "..                 ...               ...                 ...   \n",
       "118           0.000683          0.000018            0.000657   \n",
       "119           0.000682          0.000018            0.000657   \n",
       "120           0.000682          0.000019            0.000657   \n",
       "121           0.000681          0.000019            0.000657   \n",
       "122           0.000681          0.000019            0.000656   \n",
       "123           0.000681          0.000019            0.000656   \n",
       "124           0.000681          0.000019            0.000656   \n",
       "125           0.000681          0.000019            0.000656   \n",
       "126           0.000681          0.000019            0.000656   \n",
       "127           0.000681          0.000018            0.000655   \n",
       "128           0.000681          0.000019            0.000655   \n",
       "129           0.000680          0.000019            0.000655   \n",
       "130           0.000680          0.000019            0.000655   \n",
       "131           0.000680          0.000019            0.000655   \n",
       "132           0.000680          0.000019            0.000655   \n",
       "133           0.000680          0.000019            0.000655   \n",
       "134           0.000680          0.000019            0.000655   \n",
       "135           0.000680          0.000019            0.000655   \n",
       "136           0.000680          0.000019            0.000655   \n",
       "137           0.000680          0.000019            0.000655   \n",
       "138           0.000680          0.000019            0.000654   \n",
       "139           0.000679          0.000020            0.000654   \n",
       "140           0.000679          0.000020            0.000654   \n",
       "141           0.000679          0.000020            0.000654   \n",
       "142           0.000679          0.000020            0.000654   \n",
       "143           0.000679          0.000020            0.000654   \n",
       "144           0.000679          0.000020            0.000654   \n",
       "145           0.000679          0.000020            0.000654   \n",
       "146           0.000679          0.000020            0.000654   \n",
       "147           0.000679          0.000020            0.000654   \n",
       "\n",
       "     train-logloss-std  \n",
       "0             0.002588  \n",
       "1             0.001687  \n",
       "2             0.000856  \n",
       "3             0.001012  \n",
       "4             0.000507  \n",
       "5             0.001794  \n",
       "6             0.000156  \n",
       "7             0.000269  \n",
       "8             0.000466  \n",
       "9             0.000080  \n",
       "10            0.000208  \n",
       "11            0.000180  \n",
       "12            0.000129  \n",
       "13            0.000310  \n",
       "14            0.000254  \n",
       "15            0.000308  \n",
       "16            0.000295  \n",
       "17            0.000217  \n",
       "18            0.000161  \n",
       "19            0.000111  \n",
       "20            0.000099  \n",
       "21            0.000042  \n",
       "22            0.000016  \n",
       "23            0.000022  \n",
       "24            0.000018  \n",
       "25            0.000017  \n",
       "26            0.000017  \n",
       "27            0.000018  \n",
       "28            0.000024  \n",
       "29            0.000023  \n",
       "..                 ...  \n",
       "118           0.000013  \n",
       "119           0.000013  \n",
       "120           0.000013  \n",
       "121           0.000013  \n",
       "122           0.000013  \n",
       "123           0.000013  \n",
       "124           0.000013  \n",
       "125           0.000013  \n",
       "126           0.000014  \n",
       "127           0.000013  \n",
       "128           0.000013  \n",
       "129           0.000014  \n",
       "130           0.000014  \n",
       "131           0.000014  \n",
       "132           0.000014  \n",
       "133           0.000014  \n",
       "134           0.000014  \n",
       "135           0.000014  \n",
       "136           0.000014  \n",
       "137           0.000014  \n",
       "138           0.000014  \n",
       "139           0.000014  \n",
       "140           0.000014  \n",
       "141           0.000014  \n",
       "142           0.000014  \n",
       "143           0.000014  \n",
       "144           0.000014  \n",
       "145           0.000014  \n",
       "146           0.000014  \n",
       "147           0.000014  \n",
       "\n",
       "[148 rows x 4 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Best n_estimators:', 148)\n"
     ]
    }
   ],
   "source": [
    "#最佳参数n_estimators\n",
    "n_estimators = cv_result.shape[0]\n",
    "print(\"Best n_estimators:\", n_estimators)\n",
    "    \n",
    "# 采用交叉验证得到的最佳参数n_estimators，训练模型\n",
    "bst = xgb.train(param, dtrain, n_estimators)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEXCAYAAABcRGizAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcXFWd9/HPt6u6O+kkJCwdAlkk\naALihhrBbRB3cBxQB1kUH1BmGOeRcRsfB0dfOuLg4PLo6MggjAKOGyK4RAZlGFBcwQRkETEQQgyB\nQBKWkIUs3f2bP86p5KZSlapOuro63d/363VfXXf/9anu+tU5595zFRGYmZntTEe7AzAzs5HPycLM\nzBpysjAzs4acLMzMrCEnCzMza8jJwszMGnKyMGuSpH+U9JV2x2HWDk4Wo4ikiZKWSnpLYdkkScsk\nnVBYNk/SVZIek/S4pD9IOlfS3nn96ZL6Ja3L0xJJf9vi2I+WtLyV5xiMWvFExCcj4q9adL6lkl7V\nimO3wnC9X3tauYxmThajSESsA84EviCpNy/+NLAwIq4AkPRi4GfAr4BDI2IKcAzQBzyncLjfRMTE\niJgInAB8WtJzh+c3scGQVG53DDYGRISnUTYBlwLfBo4GHgEOKKz7JfBvDfY/Hfhl1bLfAm8pzB8H\n3Ak8Tko+Ty+se3pe9nje5rjCutcBfwDWAg8AHwAmAE8CA8C6PB1Y5/c6H/ivvP9NwFObKI9DgWuB\nR4FFwIm7Eg/wT8A38n4HAQG8HbgfeAx4J/AC4Pb8u3+pcJ6nAtfn92M18E1gSl739XyuJ/O5PthE\nGS8F/iGfaxNQzvMP5N9lEfDKGmXxQuAhoFRY9kbg9vz6CGAh8ATwMPC5OmV6NLC8zrrJwH8Cq4A/\nAR8BOvK6EvD/cxncB5yVy7Fc51hLgVfVWffXwOL8vs6v/M0AAj4PrATW5DJ6Zr33u93/r3vK1PYA\nPLXgTYW9gRX5H/LtheUTgH7g6Ab7n04hWeQPwMeBuXl+LrAeeDXQCXww/9N25fnFwD/m+Vfkf8xD\n8r4rgD8rxPm8/Lruh08hjkvzB8MR+cPxm8BlDfaZQPowf3ve53m5XJ4x2HionSy+DIwDXgNsBH4A\nTAWm5w+rl+Xtn5bLqxvoBX4O/Gvh2Nt9KO6sjAvb3wrMBMYDh+Tf88BCfDUTKXAv8OrC/HeBs/Pr\n3wBvy68nAi+sc4y67xcpUfwQmJTjuBs4I697J+nDekYu7/9hF5JF/rtand/PbuDfgJ/nda8Fbgam\nkBLH08lfmOq9354aT26GGoUi4jHSN9Ie4HuFVXuTmh4fqiyQ9Oncb7Fe0kcK274wL19HqlV8Hbgn\nrzsJ+K+IuDYitgCfJX1gvZj0zXUicF5EbI6I64GrgFPyvluAwyTtFRGPRcQtg/z1vhcRv42IPlKy\nOLzB9q8HlkbEJRHRl893JalpbSji+UREbIyI/yZ9uH87IlZGxAPAL4DnAkTE4lxemyJiFfA54GU7\nOe7OyrjiixFxf0Q8SfoS0J1/l86IWBoR99Y59rfJ74ekSaRv298ulMfTJO0XEesi4sbBFIakUo79\nQxGxNiKWkmoSb8ubnAh8ISKW57/T8wZz/IK3AhdHxC0RsQn4EPAiSQfl32ESqUapiLgrIlYUfr/d\neb/HLCeLUUjSqaRvdP8DfKqw6jFSc8cBlQUR8cFI/RbfJ33zrrgxIqZE6rOYBjwD+GRedyCpeaFy\njAHSt9rped39eVnFn/I6gL8kfTj9SdINkl40yF/vocLrDaTEtDNPAY7Mie9xSY+TPmimDVE8Dxde\nP1ljfiKApKmSLpP0gKQngG8A++3kuDsr44r7C+sXA+8l1X5W5nMdWOfY3wLeJKkbeBNwS0RUznUG\nqVbzR0kLJL1+JzHWsh+pRvmnwrLi+39gMe6q14NRXT7rSE180/MXlC+RmiwflnSRpL3yprv7fo9Z\nThajjKSppPbavwb+BjhR0lEAEbGe1M7/psEcMyIeJn0b/4u86EHSh3DlnCI1hzyQ182UVPzbmpXX\nERELIuJ4UlPND4DLK6cZTEyDcD9wQ058lWliRPztMMfzL/mYz46IvYBTSU0kFdXn21kZ19wnIr4V\nES/N+wXbf1EobvcH0gftscBbSMmjsu6eiDiFVB6fAq6QNKH5X5PVpG/vTyks2/r+k5qBZhTWzRzE\nsYuqy2cCsC/b/s6+GBHPJ33JmQv8v7y83vttDThZjD5fAn4QET/NVe8PAv+Rv0WS598h6eycWJA0\nA5hd74CS9iV1gt6ZF10O/LmkV0rqBP6e1Mn6a1IyWg98UFKnpKNJSeYySV2S3ippcm5aeYLUfALp\nG/m+kiYPUTlUXAXMlfS2HE+npBdIevowxzOJ1Hn9uKTp5A+vgoeBgwvzOyvjHUg6RNIr8vu8kVSr\n6a+1bfYt4N3AUaQ+i8pxTpXUm2syj+fFdY8jaVxxItVcLwfOzZdtPwV4P6kmVfm93iNpuqQppE75\nRjqrzlPO8b9d0uH5d/4kcFNELM3v75G53Nbn8uhv8H5bI+3uNPE0dBPwBtI3rilVy68Dzi3MHwlc\nTfoweBz4PXAusG9efzrpn6hyJdBKUpv21MIx3kjqqFwD3EDuMM7rnpGXrcnbvDEv7wJ+QmoOewJY\nALy0sN/FpKaEx6l/NdQ/F+aPpkGneN7uENIVVKvy8a8n9XUMKh5qd3CXC9svp3DxAOkD8iOFMrk5\nl+etpA//5YVtjweW5XN9oIkyXsr2HeLPJvUtrSVdBHBVrTIsbD+L9MH+X1XLv5Hf73WkLwdvqLP/\n0fn3r56eRuob+0Yu7/uBj7Ltaqgyqeb7COlqqPeRaiKqc56lNc7xz3ndO0md9ZXfd0Ze/krSFVDr\n2Hbl2cRG77ennU/KhWtmNuwkHQt8OSKe0nBjays3Q5nZsJE0XtLrJJVzc9zHSBdX2AjX0mQh6RhJ\niyQtlnR2jfXvVxpq4nZJ1+X2zcq6fkm35ml+K+O0PZukP9O2oUm2m9odm+1AwMdJTUG/A+4iNVPZ\nCNeyZqh8vfXdpJuKlpPaB0+JdCVGZZuXkzqlNiiNPXR0RJyU162LdNmmmZm1WStrFkcAiyNiSURs\nBi4jdeJtFemKnQ159ka2v6TOzMxGiFYOQDad7W+4WU66CqeeM4AfF+bHSVpIGuDuvIj4QfUOks4k\nDZzHhAkTnn/ooYfudtBmZmPJzTffvDoiehtt18pkoRrLarZ55TuO57H98AezIuJBSQcD10u6I6qG\nL4iIi4CLAObNmxcLFy4cmsjNzMYISX9qvFVrm6GWs/3dmTNI9wBsR2ms+g+TRibdVFkeEQ/mn0tI\nI256eGwzszZpZbJYAMyRNFtSF3AyaRjhrZSej3AhKVGsLCzfu3LHsaT9gJeQbk4yM7M2aFkzVET0\nSToLuIY0hv3FEXGnpHNID+OZD3yGdGfld9PQNyyLiONIQwpfKGmAlNDOK15FZWZmw2vU3MHtPgsz\ns8GTdHNEzGu0ne/gNjOzhpwszMysIScLMzNryMnCzMwacrIATrrwN5x04W/aHYaZ2YjlZGFmZg05\nWZiZWUNjPlk8sm4Ttyx7jFVrNzXe2MxsjBrzyaLUIbb0B/0Do+PmRDOzVhjzyaKrnIpgYJTcyW5m\n1gpjPll0llIROFeYmdU35pNFuUNI8MbnTW93KGZmI9aYTxaS6Cp1sLl/oN2hmJmNWGM+WQApWfQ5\nWZiZ1eNkQerkdrIwM6vPyYLUyb3FzVBmZnU5WeCahZlZI04WpGSxpd/XzpqZ1eNkQWqG2uSahZlZ\nXU4W5GYo91mYmdXlZAF0lcQW1yzMzOpyssA1CzOzRpwsSDfl+dJZM7P6nCxIHdy+dNbMrD4nC3yf\nhZlZI04W4IEEzcwacLLANQszs0acLPDYUGZmjThZ4JqFmVkjThb4Pgszs0acLKg0QwXhB3GbmdXk\nZAF0l1MxuHZhZlabkwXQWRKAhyk3M6ujpclC0jGSFklaLOnsGuvfL+kPkm6XdJ2kpxTWnSbpnjyd\n1so4u0q5ZuFObjOzmlqWLCSVgPOBY4HDgFMkHVa12e+AeRHxbOAK4NN5332AjwFHAkcAH5O0d6ti\n7SqXAHz5rJlZHa2sWRwBLI6IJRGxGbgMOL64QUT8NCI25NkbgRn59WuBayPi0Yh4DLgWOKZVgVaa\noVyzMDOrrZXJYjpwf2F+eV5WzxnAjwezr6QzJS2UtHDVqlW7HGhX7uD20/LMzGprZbJQjWU1e5Al\nnQrMAz4zmH0j4qKImBcR83p7e3c50EqfhZuhzMxqa2WyWA7MLMzPAB6s3kjSq4APA8dFxKbB7DtU\nKjULN0OZmdXWymSxAJgjabakLuBkYH5xA0nPBS4kJYqVhVXXAK+RtHfu2H5NXtYSlWThmoWZWW3l\nVh04IvoknUX6kC8BF0fEnZLOARZGxHxSs9NE4LuSAJZFxHER8aikT5ASDsA5EfFoq2Lt9KWzZmY7\n1bJkARARVwNXVy37aOH1q3ay78XAxa2LbputHdyuWZiZ1eQ7uCl0cLtmYWZWk5MFhQ5u1yzMzGpy\nssCXzpqZNeJkAXT60lkzs51yssADCZqZNeJkQSFZeIhyM7OanCzwHdxmZo04WVB8+JGThZlZLU4W\nQLnUQYdcszAzq8fJIusqd/g+CzOzOpwsss5Sh2sWZmZ1OFlk3a5ZmJnV5WSRdZY6PDaUmVkdThaZ\n+yzMzOpzssi63GdhZlaXk0XWWerwfRZmZnU4WWRd5Q42uWZhZlaTk0XW5ZqFmVldThZZV9l9FmZm\n9ThZZF3lDrZ41Fkzs5qcLLLOklyzMDOrw8ki6yqXfJ+FmVkdThaZaxZmZvU5WWQeG8rMrD4ni8yX\nzpqZ1edkkXmIcjOz+pwsMt9nYWZWn5NF1lnqoG8gGBjwvRZmZtWcLLKucioKd3Kbme3IySLrKqWi\ncCe3mdmOnCyyrTUL91uYme3AySJzM5SZWX0tTRaSjpG0SNJiSWfXWH+UpFsk9Uk6oWpdv6Rb8zS/\nlXFC6uAG2NLnDm4zs2rlVh1YUgk4H3g1sBxYIGl+RPyhsNky4HTgAzUO8WREHN6q+Kptq1n0D9cp\nzcz2GC1LFsARwOKIWAIg6TLgeGBrsoiIpXld29t+ukoCYLNrFmZmO2hlM9R04P7C/PK8rFnjJC2U\ndKOkNwxtaDtyn4WZWX2trFmoxrLBfG2fFREPSjoYuF7SHRFx73YnkM4EzgSYNWvWrkcKdJVKgK+G\nMjOrpZU1i+XAzML8DODBZneOiAfzzyXAz4Dn1tjmooiYFxHzent7dyvYztwM5fsszMx21MpksQCY\nI2m2pC7gZKCpq5ok7S2pO7/eD3gJhb6OVvB9FmZm9bUsWUREH3AWcA1wF3B5RNwp6RxJxwFIeoGk\n5cCbgQsl3Zl3fzqwUNJtwE+B86quohpylUtn3WdhZrajVvZZEBFXA1dXLfto4fUCUvNU9X6/Bp7V\nytiqdbtmYWZWV8OahaSnFpqEjpb0bklTWh/a8Ko0Q7nPwsxsR800Q10J9Et6GvBVYDbwrZZG1QZb\nm6FcszAz20EzyWIg9z+8EfjXiHgfcEBrwxp+lZrFBTfc22BLM7Oxp5lksUXSKcBpwFV5WWfrQmqP\nSs0ifAO3mdkOmkkWbwdeBJwbEfdJmg18o7VhDb9KB3c4W5iZ7aDh1VD5ktV3Q7r/AZgUEee1OrDh\nVnn40YCThZnZDpq5GupnkvaStA9wG3CJpM+1PrTh1dEhOgR+BLeZ2Y6aaYaaHBFPAG8CLomI5wOv\nam1Y7dEh0e9sYWa2g2aSRVnSAcCJbOvgHpVKHU4WZma1NHMH9zmkITt+FREL8iiw97Q2rPaYvd8E\nZu3T0+4wzMxGnGY6uL8LfLcwvwT4y1YG1S49XSU2bPaT8szMqjXTwT1D0vclrZT0sKQrJe0wntNo\nMKG7zIbNfe0Ow8xsxGmmz+IS0tDiB5KedPejvGzUGd/pmoWZWS3NJIveiLgkIvrydCmwe08aGqEm\ndJdZ75qFmdkOmkkWqyWdKqmUp1OBR1odWDv0dJXYsMk1CzOzas0ki3eQLpt9CFgBnEAaAmTUcc3C\nzKy2hskiIpZFxHER0RsRUyPiDaQb9Eadnq4SG7cM+F4LM7Mqu/pY1fcPaRQjxISudCXxk1vcFGVm\nVrSryUJDGsUIMb6rBMCGTW6KMjMr2tVkMSrbaSZ0p2Sx3pfPmpltp+4d3JLWUjspCBjfsojaqCc3\nQ613zcLMbDt1k0VETBrOQEYC91mYmdW2q81Qo1JPpRnKNQszs+04WRT0VDq43WdhZrYdJ4uCCe6z\nMDOrycmiwDULM7PaGj7Pos5VUWuAhcDf5+dbjAoTulNxOFmYmW2vmSflfQ54EPgW6bLZk4FpwCLg\nYuDoVgU33LrLHXQIP9PCzKxKM81Qx0TEhRGxNiKeiIiLgNdFxHeAvVsc37CSRE9XmfUeedbMbDvN\nJIsBSSdK6sjTiYV1o+5O7vRoVdcszMyKmkkWbwXeBqzM09uAUyWNB85qYWxtkYYpd83CzKyoYZ9F\n7sD+izqrfzm04bRfegCSaxZmZkUNaxaSZkj6vqSVkh6WdKWkGcMRXDtM6Cr7aigzsyrNNENdAswH\nDgSmAz/KyxqSdIykRZIWSzq7xvqjJN0iqU/SCVXrTpN0T55Oa+Z8Q6Gn230WZmbVmkkWvRFxSUT0\n5elSoLfRTpJKwPnAscBhwCmSDqvabBlwOumy3OK++wAfA44EjgA+JmlYrrzq6Sq5z8LMrEozyWK1\npFMllfJ0KvBIE/sdASyOiCURsRm4DDi+uEFELI2I24GBqn1fC1wbEY9GxGPAtcAxTZxzt/V0ld1n\nYWZWpZlk8Q7gROAhYAVwAvD2JvabDtxfmF+elzWjqX0lnSlpoaSFq1atavLQOzfBNQszsx00TBYR\nsSwijouI3oiYGhFvAN7UxLFrPXq12fsymto3Ii6KiHkRMa+3t2HLWFN6uss86WRhZradXR1I8P1N\nbLMcmFmYn0EaNqQZu7PvbpnQVWJz/wCb+6pbxszMxq5dTRa1vvlXWwDMkTRbUhdpTKn5TR7/GuA1\nkvbOHduvyctabnzlaXmuXZiZbbWryaJhc1JE9JHu8L4GuAu4PCLulHSOpOMAJL1A0nLgzcCFku7M\n+z4KfIKUcBYA5+RlLTchD1O+3pfPmpltVfcO7jpDk0OqVYxv5uARcTVwddWyjxZeLyA1MdXa92LS\nqLbDqmfrMOVOFmZmFXWTRURMGs5ARooJfgCSmdkO/KS8Kj1bH63qZGFmVuFkUWVCd6Vm4WYoM7MK\nJ4sqPVs7uF2zMDOrcLKoUmmG8pAfZmbbOFlUmVBJFq5ZmJlt5WRRZXyX+yzMzKo5WVTpKncg4Js3\nLWt3KGZmI4aTRQ2lDtE/0OyYh2Zmo1/DZ3CPRU+bOpGZ+/S0OwwzsxHDNYsaeid1s2rtpnaHYWY2\nYjhZ1OBkYWa2PSeLGirJIsL9FmZm4GRRU+/Ebjb3D/DEk7581swMnCxqmrrXOABWrt3Y5kjMzEYG\nJ4saeid2A7jfwswsc7KoYepeKVmsdLIwMwOcLGrqneSahZlZkZNFDZO6y3SXO9xnYWaWOVnUIImp\ne/leCzOzCieLOnondrNqnZOFmRk4WdQ1ddI4Vj7hZGFmBk4WdfVOcs3CzKzCyaKO3kndPL5hC5v6\n/MQ8MzMnizqm5stnV6/b3OZIzMzaz8miDt9rYWa2jZNFHVMn5fGhnvC9FmZmThZ1bK1ZuJPbzMzJ\nop59J3Yh4ctnzcxwsqirs9RBSeLbv13W7lDMzNrOyWInujs72LjFl86amTlZ7MTrn3UgnSUXkZmZ\nPwl3Yu60STyyfjOr3cltZmNcS5OFpGMkLZK0WNLZNdZ3S/pOXn+TpIPy8oMkPSnp1jx9uZVx1nPI\n/pMAuPvhte04vZnZiNGyZCGpBJwPHAscBpwi6bCqzc4AHouIpwGfBz5VWHdvRByep3e2Ks6dmTtt\nIgB3P+RkYWZjWytrFkcAiyNiSURsBi4Djq/a5njga/n1FcArJamFMQ1K78RupvR0sujhde0Oxcys\nrVqZLKYD9xfml+dlNbeJiD5gDbBvXjdb0u8k3SDpz2qdQNKZkhZKWrhq1aqhjT4dn7n7T3IzlJmN\nea1MFrVqCNHkNiuAWRHxXOD9wLck7bXDhhEXRcS8iJjX29u72wHXcsj+k7j7obVEVIduZjZ2tDJZ\nLAdmFuZnAA/W20ZSGZgMPBoRmyLiEYCIuBm4F5jbwljrmjttEms39bFijceIMrOxq5XJYgEwR9Js\nSV3AycD8qm3mA6fl1ycA10dESOrNHeRIOhiYAyxpYax1Va6IWuSmKDMbw1qWLHIfxFnANcBdwOUR\ncaekcyQdlzf7KrCvpMWk5qbK5bVHAbdLuo3U8f3OiHi0VbHuzNz9fUWUmVm5lQePiKuBq6uWfbTw\neiPw5hr7XQlc2crYmjWlp4v99+p2zcLMxjTfwd2EjVv6+cnvH2p3GGZmbeNk0YQ3P38m/QNB/4Cv\niDKzscnJoglzp01iU98Ayx7d0O5QzMzawsmiCVuviHInt5mNUU4WTZhTuSLKndxmNkY5WTShp6vM\nrH16fEWUmY1ZThZNmpuH/TAzG4ucLJp0yLSJ3Ld6PZv6/JhVMxt7nCyaNHf/SfQNBCdc8Ot2h2Jm\nNuycLJp0yLR0RdSTm12zMLOxx8miSQfvNxEBG5wszGwMcrJoUle5g+7ODjZscbIws7HHyWIQJnaX\nWbexz8N+mNmY42QxCFPGd9I3ENy2/PF2h2JmNqycLAZh8vhOAH62aOif921mNpI5WQxCudTBxO4y\nNyxa2e5QzMyGlZPFIHznb17EmUcdzO0PrOGRdZvaHY6Z2bBxshikow/pJQJ+cc/qdodiZjZsnCwG\n6ZkHTqbcIT5x1R/aHYqZ2bBxshikjg6xd08Xj27YzNLV69sdjpnZsHCy2AUz9h5PB+LDP7iDCN9z\nYWajn5PFLvj+u17Cx49/Br9a/AhX3vJAu8MxM2s5J4td9JYjZjGxu8zZV97Oal8ZZWajnJPFLuro\nELP3m0D/QPDP7uw2s1HOyWI3/OjvXsrfvXIOP7j1QW6423d1m9no5WSxm9718qcyrrODv/raAm73\nmFFmNko5Weym7nKJK975YnondvOmf/81//HzJb5CysxGHSeLIfDM6ZP58XuOYtK4MudefRdf/eV9\n7Q7JzGxIOVkMkck9ndz8kVfzumdN49yr7+LHd6xod0hmZkPGyWIIdXSIz514OBO6yvzfb93CBT+7\nly39A+0Oy8xstzlZDLFxnSV+8cGX89rDpvGpn/yR13/xl3zjxj/x6PrN7Q7NzGyXabR0xs6bNy8W\nLlzY7jC2c82dD/HZaxZxz8p1lDvEUXN7+YvnHMDzZ+2ThgzpULtDNLMxTtLNETGv0XblFgdxDPAF\noAR8JSLOq1rfDfwn8HzgEeCkiFia130IOAPoB94dEde0MtZWeO0zpvGaw/bnrhVr+eFtD/CjWx/k\n+j+mByeN7ywxZ/+JzJk6iZn7jOeAyeOYNrnycxx7jetsc/RmZtu0LFlIKgHnA68GlgMLJM2PiOLt\nzmcAj0XE0ySdDHwKOEnSYcDJwDOAA4H/kTQ3IvpbFW+rSOKwA/fisAP34h9eeyh3PLCGPz70BIse\nWsfdD6/lF/esYuXaHYcLmdhdZtrkcRwweRz7TOhiYneZiePKTOzKP7vLW5f1dJXpLnfQWeqgsyQ6\nSx10Vc13ljoouSZjZruolTWLI4DFEbEEQNJlwPFAMVkcD/xTfn0F8CVJyssvi4hNwH2SFufj/aaF\n8bZcR4d4zswpPGfmlO2Wb+rrZ+UTm3joiY2sWLORh9Y8mX9u5ME1G1n26AbWbexj7aY+Nvfteod5\nh6DUISTRIeiQ6JDQ1tdsna+3jQTVKSe9ZVXLGi6ouWiHY9Xepvo4Nc7fRF5sJu5ax9nV8+9YJk2U\n2xBrplx26/itPHaLg9+Ty37O/pP45Buf1boT0NpkMR24vzC/HDiy3jYR0SdpDbBvXn5j1b7TWxdq\ne3WXS8zcp4eZ+/Q03HZz3wDrN/WxrjCt39THlv5gS/8AW/oH2Nw3sP18/wBb+tL8QAQDARGx9fVA\nBJF/FtdvW7b9NkW1uryqF9XqF6vZUxbVszX2q96m5vkHH2Pt7Zo4f83jNP59m41pKO3J/ZOtDr3W\n39qQHr/F8Zda/S2A1iaLWtFXF1m9bZrZF0lnAmcCzJo1a7Dx7ZG6yh10lbvYe0JXu0MxszGklZfO\nLgdmFuZnAA/W20ZSGZgMPNrkvkTERRExLyLm9fb2DmHoZmZW1MpksQCYI2m2pC5Sh/X8qm3mA6fl\n1ycA10eqK88HTpbULWk2MAf4bQtjNTOznWhZM1TugzgLuIZ06ezFEXGnpHOAhRExH/gq8PXcgf0o\nKaGQt7uc1BneB7xrT7wSysxstPBNeWZmY1izN+V5uA8zM2vIycLMzBpysjAzs4acLMzMrKFR08Et\naRXwp904xH7A6iEKp1Uc49DZE+LcE2KEPSPOPSFGaE+cT4mIhjeqjZpksbskLWzmioB2coxDZ0+I\nc0+IEfaMOPeEGGFkx+lmKDMza8jJwszMGnKy2OaidgfQBMc4dPaEOPeEGGHPiHNPiBFGcJzuszAz\ns4ZcszAzs4acLMzMrKExnywkHSNpkaTFks5udzwVkmZK+qmkuyTdKek9efk+kq6VdE/+ufcIiLUk\n6XeSrsrzsyXdlGP8Th6ivp3xTZF0haQ/5vJ80Qgtx/fl9/r3kr4taVy7y1LSxZJWSvp9YVnNslPy\nxfy/dLuk57U5zs/k9/x2Sd+XNKWw7kM5zkWSXtuuGAvrPiApJO2X59tWlvWM6WQhqQScDxwLHAac\nIumw9ka1VR/w9xHxdOCFwLtybGcD10XEHOC6PN9u7wHuKsx/Cvh8jvEx4Iy2RLXNF4CfRMShwHNI\nsY6ocpQ0HXg3MC8inkka1v9k2l+WlwLHVC2rV3bHkp49M4f0BMsLhilGqB3ntcAzI+LZwN3AhwDy\n/9HJwDPyPv+ePwvaESOSZgKvBpYVFrezLGsa08kCOAJYHBFLImIzcBlwfJtjAiAiVkTELfn1WtIH\n3HRSfF/Lm30NeEN7IkwkzQDYvYaQAAAF1klEQVT+HPhKnhfwCuCKvElbY5S0F3AU6dkpRMTmiHic\nEVaOWRkYn58a2QOsoM1lGRE/Jz1rpqhe2R0P/GckNwJTJB3Qrjgj4r8joi/P3kh64mYlzssiYlNE\n3AcsJn0WDHuM2eeBD7L9o6PbVpb1jPVkMR24vzC/PC8bUSQdBDwXuAnYPyJWQEoowNT2RQbAv5L+\n0Afy/L7A44V/0naX6cHAKuCS3FT2FUkTGGHlGBEPAJ8lfbtcAawBbmZklWVFvbIbyf9P7wB+nF+P\nmDglHQc8EBG3Va0aMTFWjPVkoRrLRtS1xJImAlcC742IJ9odT5Gk1wMrI+Lm4uIam7azTMvA84AL\nIuK5wHpGRtPddnK7//HAbOBAYAKpKaLaiPr7rDLS3nsAJH2Y1Kz7zcqiGpsNe5ySeoAPAx+ttbrG\nsraW5VhPFsuBmYX5GcCDbYplB5I6SYnimxHxvbz44Up1NP9c2a74gJcAx0laSmrCewWppjElN6VA\n+8t0ObA8Im7K81eQksdIKkeAVwH3RcSqiNgCfA94MSOrLCvqld2I+3+SdBrweuCtse2mspES51NJ\nXw5uy/9DM4BbJE1j5MS41VhPFguAOfmKky5Sp9f8NscEbG37/ypwV0R8rrBqPnBafn0a8MPhjq0i\nIj4UETMi4iBS2V0fEW8FfgqckDdrd4wPAfdLOiQveiXp2e4jphyzZcALJfXk974S54gpy4J6ZTcf\n+D/5Sp4XAmsqzVXtIOkY4B+A4yJiQ2HVfOBkSd2SZpM6kX873PFFxB0RMTUiDsr/Q8uB5+W/2RFV\nlgBExJiegNeRrpS4F/hwu+MpxPVSUrXzduDWPL2O1CdwHXBP/rlPu2PN8R4NXJVfH0z651sMfBfo\nbnNshwMLc1n+ANh7JJYj8HHgj8Dvga8D3e0uS+DbpD6ULaQPszPqlR2p6eT8/L90B+nKrnbGuZjU\n7l/5//lyYfsP5zgXAce2K8aq9UuB/dpdlvUmD/dhZmYNjfVmKDMza4KThZmZNeRkYWZmDTlZmJlZ\nQ04WZmbWkJOFmZk15GRhthskHS7pdYX54zREQ91Lem8eEsKs7XyfhdlukHQ66Yaps1pw7KX52KsH\nsU8pIvqHOhYz1yxsTJB0kNKDj/4jP2DovyWNr7PtUyX9RNLNkn4h6dC8/M1KDya6TdLP8xAx5wAn\nSbpV0kmSTpf0pbz9pZIuUHqI1RJJL8sPwLlL0qWF810gaWGO6+N52btJAwr+VNJP87JTJN2RY/hU\nYf91ks6RdBPwIknnSfpDfmjOZ1tTojbmtPsWck+ehmMCDiKNPHp4nr8cOLXOttcBc/LrI0ljXkEa\ndmF6fj0l/zwd+FJh363zpIfdXEYauuF44AngWaQvaTcXYqkMl1ECfgY8O88vZdvwDweSxo/qJY2k\nez3whrwugBMrxyINYaFinJ487e7kmoWNJfdFxK359c2kBLKdPCT8i4HvSroVuBCoPHTmV8Clkv6a\n9MHejB9FRJASzcORBo8bAO4snP9ESbcAvyM9va3W0xpfAPws0qi0leG2j8rr+kmjE0NKSBuBr0h6\nE7BhhyOZ7YJy403MRo1Nhdf9QK1mqA7SA4cOr14REe+UdCTpyYC3Stphm52cc6Dq/ANAOY96+gHg\nBRHxWG6eGlfjOLWeb1CxMXI/RUT0STqCNGrtycBZpKHjzXaLaxZmBZEeMHWfpDdDGipe0nPy66dG\nxE0R8VFgNel5A2uBSbtxyr1ID2RaI2l/tn/gUfHYNwEvk7Rffl70KcAN1QfLNaPJEXE18F7SiLtm\nu801C7MdvRW4QNJHgE5Sv8NtwGckzSF9y78uL1sGnJ2brP5lsCeKiNsk/Y7ULLWE1NRVcRHwY0kr\nIuLlkj5Eer6FgKsjotazLSYBP5Q0Lm/3vsHGZFaLL501M7OG3AxlZmYNuRnKxixJ55OeI170hYi4\npB3xmI1kboYyM7OG3AxlZmYNOVmYmVlDThZmZtaQk4WZmTX0v9Su8tLrpIEPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113d085d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot\n",
    "test_means = cv_result['test-logloss-mean']\n",
    "test_stds = cv_result['test-logloss-std'] \n",
    "\n",
    "x_axis = range(0, cv_result.shape[0])\n",
    "        \n",
    "pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "\n",
    "pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "pyplot.xlabel( 'n_estimators' )\n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 100.00%\n"
     ]
    }
   ],
   "source": [
    "# make prediction\n",
    "preds = bst.predict(dtest)\n",
    "y_pred = [round(value) for value in preds]\n",
    "y_test = dtest.get_label()\n",
    "test_accuracy = accuracy_score(y_test, y_pred)\n",
    "print(\"Test Accuracy: %.2f%%\" % (test_accuracy * 100.0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用XGBoost内嵌的函数，按特征重要性排序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmYFOW5/vHvw6IiiyiCAoPgAsgi\nEFDQqCwxoIIeJRqVExcUEqMxYoxxiUnAJJ5Exbgdc4xr1PjToOISY1wRSRQioGBQg0aZCAoCKoEZ\nQLb798dbDc3YM90w01PVxfO5rrnorq6ueh8noaiqu57XJOGcc87l0yDuATjnnCsNfsBwzjlXED9g\nOOecK4gfMJxzzhXEDxjOOecK4gcM55xzBfEDhnN1wMxuM7Ofxj0O54rJ/DkMFyczKwf2AjZmLe4i\n6eNabHMw8AdJZbUbXWkys98DiyT9JO6xuHTxMwyXBMdLapb1s90Hi7pgZo3i3H9tmFnDuMfg0ssP\nGC6xzOxQM3vVzFaY2dzozCHz2dlm9o6ZrTKzD8zs3Gh5U+AvQDszq4h+2pnZ783sl1nfH2xmi7Le\nl5vZZWb2JlBpZo2i7z1qZsvMbIGZXVjDWDdvP7NtM7vUzJaa2WIzO9HMhpvZu2b2mZn9OOu7E8zs\nETP7Y1TP62bWO+vzbmY2Nfrv8JaZ/VeV/f6fmT1tZpXAGOBbwKVR7X+K1rvczN6Ptv+2mY3M2sZo\nM/ubmU00s8+jWo/N+nwPM7vHzD6OPn8867PjzGxONLZXzaxXwb9gV3L8gOESyczaA38GfgnsAVwC\nPGpmraNVlgLHAS2As4EbzKyvpErgWODj7ThjGQWMAFoCm4A/AXOB9sBRwEVmdnSB29ob2CX67s+A\nO4DTgX7AkcDPzGy/rPVPAB6Oav1/wONm1tjMGkfjeA5oA3wfeMDMumZ997+Bq4HmwH3AA8C1Ue3H\nR+u8H+13N+Aq4A9m1jZrGwOA+cCewLXAXWZm0Wf3A7sCPaIx3ABgZn2Bu4FzgVbA74AnzWznAv8b\nuRLjBwyXBI9H/0JdkfWv19OBpyU9LWmTpOeBWcBwAEl/lvS+gpcJf6EeWctx3CxpoaQ1wCFAa0k/\nl7RO0geEv/RPK3Bb64GrJa0HHiL8RXyTpFWS3gLeArL/NT5b0iPR+r8hHGwOjX6aAb+OxjEFeIpw\ncMt4QtIr0X+ntbkGI+lhSR9H6/wReA/on7XKvyXdIWkjcC/QFtgrOqgcC3xX0ueS1kf/vQG+DfxO\n0t8lbZR0L/BFNGaXQiV7rdalyomSXqiyrCPwTTM7PmtZY+AlgOiSyXigC+EfPrsC/6jlOBZW2X87\nM1uRtawh8NcCt/Vp9JcvwJroz0+yPl9DOBB8ad+SNkWXy9plPpO0KWvdfxPOXHKNOyczOxO4GOgU\nLWpGOIhlLMna/+ro5KIZ4YznM0mf59hsR+AsM/t+1rKdssbtUsYPGC6pFgL3S/p21Q+iSx6PAmcS\n/nW9PjozyVxCyRX9qyQcVDL2zrFO9vcWAgskdd6ewW+HDpkXZtYAKAMyl9I6mFmDrIPGPsC7Wd+t\nWu9W782sI+Hs6ChguqSNZjaHLf+9arIQ2MPMWkpakeOzqyVdXcB2XAr4JSmXVH8Ajjezo82soZnt\nEt1MLiP8K3ZnYBmwITrbGJb13U+AVma2W9ayOcDw6Abu3sBFefb/GrAyuhHeJBpDTzM7pM4q3Fo/\nM/tGlNC6iHBpZwbwd8LB7tLonsZg4HjCZa7qfAJk3x9pSjiILIMQGAB6FjIoSYsJIYLfmtnu0RgG\nRh/fAXzXzAZY0NTMRphZ8wJrdiXGDxgukSQtJNwI/jHhL7qFwI+ABpJWARcCk4DPCTd9n8z67j+B\nB4EPovsi7Qg3bucC5YT7HX/Ms/+NhL+Y+wALgOXAnYSbxsXwBHAqoZ4zgG9E9wvWAf9FuI+wHPgt\ncGZUY3XuArpn7glJehu4HphOOJgcBLyyDWM7g3BP5p+EsMFFAJJmEe5j/G807n8Bo7dhu67E+IN7\nzsXMzCYAB0g6Pe6xOFcTP8NwzjlXED9gOOecK4hfknLOOVcQP8NwzjlXkFQ9h9GyZUsdcMABcQ+j\nTlRWVtK0adO4h1FraakDvJakSkstcdYxe/bs5ZJa51svVQeMvfbai1mzZsU9jDoxdepUBg8eHPcw\nai0tdYDXklRpqSXOOszs34Ws55eknHPOFcQPGM45V2LWrl1L//796d27Nz169GD8+PEALFiwgAED\nBtC5c2dOPfVU1q1bV6f7jeWAYWYXWpjL4HMzezPqpz/LzI7IWmcfM3suWu9tM+sUx1idcy5pdt55\nZ6ZMmcLcuXOZM2cOzzzzDDNmzOCyyy7jBz/4Ae+99x677747d911V53uN64zjPMJbao7AL0l9QHO\nIbReyLgPuE5SN0Ib5qX1PkrnnEsgM6NZs9DseP369axfvx4zY8qUKZx88skAnHXWWTz++OM1bWab\n1fsBw8xuIzRGexL4trY8CJJpkIaZdQcaRXMgIKlC0ur6HqtzziXVxo0b6dOnD23atGHo0KHsv//+\ntGzZkkaNQpaprKyMjz76qE73GcuDe2ZWDhwsaXk0VeSvCDN5jZA03cxOBMYC64B9gReAy7PmF8je\n1neA7wDsuWfrfj+78Y56qqK49moCn6zJv17SpaUO8FqSKi217Ltbw81nDduioqKCn/70p5x99tlc\nc801PPDAAwAsXbqUyy+/nLvvvjvvNoYMGTJb0sF5V5RU7z+EjqF7Vlk2EHghen0y8B/CmUgjwtwH\nY/Jtt0uXLkqLl156Ke4h1Im01CF5LUmVllpqU8eECRN07bXXqlWrVlq/fr0k6dVXX9WwYcMK+j4w\nSwX83Z2YlJSkacD+ZrYnsAh4Q9IHkjYAjwN9Yx2gc84lxLJly1ixIsxntWbNGl544QW6devGkCFD\neOSRRwC49957OeGEE+p0v7EeMMzsgMxE89GE8jsBnwIzgd3NLPPk4deAt+MZpXPO1c7ChQsZMmQI\n3bp1o0ePHtx0000AzJkzh0MPPZQ+ffpw7rnn8tprrxW0vcWLFzNkyBB69erFIYccwtChQznuuOO4\n5ppr+M1vfsMBBxzAp59+ypgxY+q0jrie9G4OvArsBTQzs3XAJuDB6PRoo5ntBXwYHU8qgXNjGqtz\nztVKo0aNuP766+nbty+rVq2iX79+DB06lEsvvZTx48dz7LHH8utf/5pLL72UqVOn5t1er169eOON\nN760fL/99iv4oLM94jpgLCPMINYRuETScTnWWQv0lLS8XkfmnHN1rG3btrRt2xaA5s2b061bNz76\n6CPMjJUrVwKhl1S7du3iHGZe9X7AqBKrzX/73jnnUqS8vJw33niDAQMGcOONN3L00UdzySWXsHbt\n2sT3wos1VkuYiP5Rwk3ujwlnG29F6ywgzBMs4HeSbq9mWx6rTbC01AFeS1LFVctB7bd9evc1a9Yw\nbtw4Tj/9dAYOHMjNN99M7969GTRoEH/5y1944YUXuP7664sw2pqVRKwWaAE0i5YNB97LWqdd9Gcb\nYC4wMN92PVabPGmpQ/JakqpUalm3bp2GDRum66+/fvOyFi1aaNOmTZKkKVOmqHnz5rGMjVKI1Upa\nKakiev000DiK1SLp4+jPpcBjhPYgzjlXciQxZswYunXrxsUXX7x5ebt27Xj55ZcBeP311+ncuXNc\nQyxI3LHavbNitf2j8XxqZk3NrHm0vCkwDJgX30idc/lUFx19+OGH6dGjBw0aNEj8NfpieeWVV7j/\n/vuZMmUKffr0oU+fPjz99NPccccd/PCHP6R3797ceeed3H57zivviVG0m95mdiFwHuH5iXaEB++u\nlDQxa7XHgEPMbD3hstNpkmRmNwBnmpmA9cBNkp4p1lidc7VXXXS0Z8+eTJ48mXPP3XGT8UcccUTm\nUvuXzJ49GwgTKPXr168+h7XNipmSOp8Qna0kxGdPzHwgqROAmV0GVAD3STo067uPAOdL2mBm1xAe\n6HPOJVh10dGhQ4fGPDJXV4pySapKdPZbkmYSzhS2otAO5LMcy59TaAkCMAMoK8Y4nXPFkR0ddelR\ntFhtdkfa6P0EoKLKJSmiiZGektSzmu38CfijpD9U87nHahMsLXXAjllLXURHMy666CLOO+88unbt\nus3brElFRcV2dXlNmjjriD1WS5WOtMAEwnMWVdfrBMyrZhtXEu5zWCH79Fht8qSlDslrKUSu6GjG\noEGDNHPmzDrfZ1p+L3HWQYGx2rhag+RlZmcBxwFHRQU55xJM1URHXXokpr15NjM7BrgM+C/5THvO\nFcU555xDmzZt6Nlzy9XguXPncthhh3HQQQdx/PHHb+5zVIjqoqOPPfYYZWVlTJ8+nREjRnD00UcX\noxxXD4p+hmFmewOzCE91bzKzi4CbCHN4HwhsABqZ2RfA/0i6CvhfoBVbutW+LunwYo/VuR3J6NGj\nueCCCzjzzDM3Lxs7diwTJ05k0KBB3H333Vx33XX84he/KGh7NUVHR44cWSdjdvEq2hmGpE6Slkta\nIqlMUgtJLSWVAWMIrUAOB9pIMkLsdnj09ROBjwgHjebAGjNL9iOQzpWYgQMHsscee2y1bP78+Ztv\nVA8dOpRHH300jqG5hKr3S1JVIrcDJH0efZQdn+0GzJC0WiFe+zLg/0Rxrsh69uzJk08+CYQntBcu\nXBjziFySxNqtVllzXZjZJcCBksaaWTfgCeAwYA3wIuEu/vdzbMtjtQmWljqgNGopNAabiXAuWbKE\nK664gnvuuQeADz/8kFtuuYX//Oc/HH744UyePJknnniimEOuNY/V1l7ssdqafvhy5HYI8A7QKmvZ\nGOB1YBpwG3BDvu16rDZ50lKHlM5aFixYoB49euRcZ/78+TrkkEPqcVTbJy2/l1KI1caekjKzXsCd\nwAmSPs0sl3SXpL6SBhKeBn8vrjE6t6NYunQpAJs2beKXv/wl3/3ud2MekUuSuLvV7gNMBs6Q9G6V\nz9pkrfMN4MH6H6FzW8sVRc2YOHEiZsby5aUxq/CoUaM47LDDmD9/PmVlZdx11108+OCDdOnShQMP\nPJB27dpx9tlnxz1MlyBFjdVW17E2a5WfAR2Al81sA/CWtlxHe8bMDgQM+BDoDBRvdnPnCpArigqh\ntffzzz/PPvvsE9PItt2DD+b+N9i4cePqeSSuVBT7DON8QlT2POBCYCJsFbkdCwwFBgDva+ubLsuB\nkyQ1AX4AXFvksTqXV64oKsAPfvADrr32WqLnhpxLpaIdMGrbsZYwl3eL6PVuhDm/nUucJ598kvbt\n29O7d++4h+JcURXtkpSk70YtPoYoKz67DS4CnjWziYQD21dzrVQlVsstDyQ7AliovZqQilpKoY5t\niaJOnTqVJUuWUFlZydSpU1m7di2XXXYZ11133eb3r7zyCrvttu1dXutTppY0SEstJVFHIVGq7f2h\nFh1rgZsJl6QATgFeyLc/j9UmT1rqkHJHUd988021bt1aHTt2VMeOHdWwYUN16NBBixcvjnGk+aXx\n91LqSiFWm9hutcBZQObu28OE6K1ziXLQQQdtjqICdOrUiVmzZrHnnnvGOCrniiP25zBq8DEwKHr9\nNfw5DFdEueKyP/3pT+nVqxd9+vRh2LBhLF++PGcU1bkdRb2cYeToWPsT4BOgC/AfoCnQ0Mw+AX4s\n6S7gfeD5KG77JtF9CueKIVdc9kc/+tHmTq0333wz99133+Y+S9UpLy8v5jCdi1VRzzBUTcdaYAlw\nDKFb7b6SGgMjgPLoYAEhRpuJ2w6QNLuYY3U7tlxx2RYtWmx+XVlZ6ZFZt8NLarfamuK2ztWbK6+8\nkg4dOvDAAw/4U89uh5fIbrVZyzoBT0n6ch+GLet4t9oEi7OOQuOyGVU7t2Z74IEHqKio4Nxzz62r\n4cUqLR1eIT21eLfaWnSrVTVx25p+PFabPKVUR02dW8vLy9WpU6d6HlHxlNLvJZ+01FIKsdrYU1LV\ndat1Lm7vvbclmPfkk0+WVJ8o54ohsd1qnatJrhjsww8/TI8ePWjQoAGzZs3apu3listefvnl9OzZ\nk169evHcc89xwQUX1HUZzpWUuB7caw68CrQHdgVejBIon0lqD2Bm7xMuSTUws0XAeG1JULkdXK4Y\nbM+ePZk8efJ23WfI1bl1zJgxW71PfNsG54osrgPGMuBYoCOhVchxOdY5G6gA7lMNN73djmngwIFf\neuahW7du8QzGuR1E3LHar1S3njxW65xziVLvZxjK6mIL9AR+YmZzCa1ALpH01rZsz7vVJtu21LE9\nMdhM19hsK1asYPbs2VRUVGzT9vIpiW6iBfJakqck6igkSlXXP0SxWkKrkGbRsuHAe1XW64THakta\nMeuoLgY7aNAgzZw5s873l5bfieS1JJHHavOQtFJSRfT6aaCxmXmbT+ecS6C4Y7V7WxSPMrP+0Xj8\nWYwScsMNN9CjRw969uzJqFGjWLt2bb3sN1cM9rHHHqOsrIzp06czYsQIjj766HoZi3M7irhjtWWE\n2KyAjYR7GAIws7eBzkAjM/sUuFQeq02Ujz76iJtvvpm3336bJk2acMopp/DQQw8xevToou87VwwW\nYOTIkUXft3M7qrjOMJYBRwNtgCaSmhCmYL0IwMxGAB8BTYBmwALCJEouYTZs2MCaNWvYsGEDq1ev\npl27dnEPyTlXJHHHar+dOaMgzImRed0deFnSBkmVwFxCO3SXIO3bt+eSSy5hn332oW3btuy2224M\nGzYs7mE554ok9m61ZjYS+BXhbGOEpOlmNgwYDwwlPAn+GnCrpOtzbMu71dahbYm2rlq1ivHjx/Oz\nn/2MZs2aMWHCBAYNGsTQoUM3r5OWTqLgtSRVWmrxbrUFdquNlg0EXsh6fyUwB3geeAAYl2+7Hqut\nX5MmTdI555yz+f29996r8847b6t1SqGOQnktyZSWWjxWuw0UnuzePxOrlXS1pD6ShgKGz+mdOPvs\nsw8zZsxg9erVSOLFF1/09hzOpVjcsdoDsmK1fYGdgE/NrKGZtYqW9wJ6Ac/FN9L0WLFiBSeffDIH\nHngg3bp1Y/r06du9rQEDBnDyySfTt29fDjroIDZt2sR3vuNTrzuXVkWN1ZrZhcB5wNtAO6Av4VJT\nxknA+Wa2F+GG9+8lycwaA7Ozlr9QzHHuSMaNG8cxxxzDI488wrp161i9enWttnfVVVdx1VVX1dHo\nnHNJVuwzjPMJLT/OAy4EJgJI6qQwPetEYAMhFbUb8FUz6w6sAxoCvSXtSriXcVaRx5p6K1euZNq0\naZvbdu+00060bNky5lE550pF0Q4YVeKz35I0E1hfZbX+wL8kfSBpHfAQcALQCvhCWyZVep5wNuJq\n4YMPPqB169acffbZfOUrX2Hs2LFUVlbGPSznXIkoaqw2Oz4bvZ8AVEiaGL0/GThG0tjo/RnAAOD7\nhCTVSZJmmdlNwNckHZRjHzt0rHZbYrDz58/n/PPP55ZbbqF79+7ccsstNG3alHPOOacWI61ZWiKP\n4LUkVVpq2eFjtVSJzwITCO0/Mu+/CdyZ9f4M4Jbo9WHAXwnPYPwSeCPf/jxWW7PFixerY8eOm99P\nmzZNw4cPr/P9ZEtL5FHyWpIqLbV4rDa/RUCHrPdlhHkxkDRd0pGS+gPT8Fhtre2999506NCB+fPn\nA/Diiy/SvXv3mEflnCsVcTUfzJgJdDazfQm9o04D/hvAzNpIWmpmOwOXAVfHN8z4bNy4kYMPPpj2\n7dvz1FNP1Xp7t9xyC9/61rdYt24d++23H/fcc08djNI5tyOolwOGme0NzCJMmLTJzH4CfAJ0IVy2\n+ichPnuHtsy49yMzOw7YF3hf0pT6GGvS3HTTTXTr1o2VK1fWyfb69OnDrFmz6mRbzrkdS1EvSSmK\nz0paIqlMUgtJLYElhGaChwN9Je0MjCSkpjLf/RFwBzCZ0K12h7No0SL+/Oc/M3bs2LiH4pxzsXer\nHSDp8+ijGYR7GJn1yoARwJ31PcakuOiii7j22mtp0CDuW03OORfDPQxJ3zWzY4AhiuK2kTHAX7Le\n3whcSphsqSBr1m+k0+V/rpuBFkn5r0cUtN706dNp06YN/fr1S/7E8M65HULs7c2j90OA3wJHSPo0\nuncxXNL5ZjaYEMU9rpptbX4Oo3Xr1v0mTZpUHyUU3a233srUqVNp2LDh5hYeRx55JFdeeWX+LydI\nWjLy4LUkVVpq2eGfw6juh6znMwiNBd8HumR9/itC5LaccL9jNfCHfNtN63MYL730kkaMGBHfYGoh\nLRl5yWtJqrTUUgrPYcQaqzWzfQg3tc/QljYgSLoCuCJaZzDhDOP0WAYZs40bNzJ27FhWrVoV91Cc\nczu4uA4YzYFXgc7ARuBFM9sElEvqYWa7EB7W25kQxa3ag2qHcdNNN9G/f/86i9U659z2iit+sww4\nmhCrbS2pCXAyUBF9/gWhd1RvwrMan5vZobGMNEYeq3XOJUkiY7XRZbXMwaNx9FP/d+dj5rFa51yS\nJDZWa2YNgdnAAcCtkv6eb9seq3XOueJJZKy2yrotgceA70ual2NbHqtNsLREHsFrSaq01OKx2u2M\n1eZYfzxZbdGr+/FYbfKkJfIoeS1JlZZaSiFWG+vF8epitWbWOjqzwMyaAF8nNCgseWvXrqV///70\n7t2bHj16MH78+LiH5JxzBSnqAcPMLjSzd8zsUTObbmZfmNklWav8DNgLmBJ9tiha3hb4wMzWAp8D\nTYFniznW+rLzzjszZcoU5s6dy5w5c3jmmWeYMWNGjd8ZPHhwnbQ2d8652ij2Gcb5wHDgPOBCYCJs\n6WILnAssBboTns341My6S3oTOB1oEv3MB1KRLTWzzdcp169fz/r16zGzmEflnHP5Fe2AUSU++y1J\nM/nyA3j9gX9J+kDSOuAh4AQASU9nXV97jaxOtqVu48aN9OnThzZt2jB06FAGDBgQ95Cccy6vosVq\nVX18Nlt7YGHW+0XAVn97mlljwlzf4/LtM65YbaFR2YyGDRsyZ84cVqxYwciRI5k3bx49e/Ys0uic\nc65uxD1Fa65rMVVzvr8Fpkn6a84NbB2rZdIxTet2hAWozXMSnTp14tZbb+XUU0/danlFRUUqnr9I\nSx3gtSRVWmopiToKiVJt7w9Z8dno/QSy4rHAYcCzWe+vAK7Iej8eeBxoUMj+SiFWu3TpUn3++eeS\npNWrV+uII47Qn/70py+t51HB5PFakikttXisNr+ZQGcz29fMdgJOI9zzwMzGEvpNjZK0KcYx1qny\n8nLat29PkyZN2H333WnYsCHHHZdzqg/nnEuUYh8wmgOvmtlTZvYF4YzhF2a2yMxaSNoA3ENIQVUC\nSyS9FX33NqAbofHgWjNLds+PAh188MF88sknrFmzhlWrVrFmzZq8sVrnnEuCYt/DWAYcSzgYdARO\nBD6XNBE294saDRxIuOE9M4rVvg18GxgCjJa0yczaFHms9cJjtc65UrXNZxhmtruZ9SpgvVrFagnP\nbvw8czlK0tJtHWtSeazWOVeKCjrDMLOpwH9F688BlpnZy5Iuru47qn2sdn/gVDMbSThTuVDSezWN\n02O1zjlXPIVektpN0sroRvQ9ksab2Zt1sP+aYrU7A2slHWxm3wDuBo780gY8VptoaakDvJakSkst\nJVFHIVEq4B+E/k7PAYdEy94s4HvlbGesltBssFP02oD/5Nufx2qTJy11SF5LUqWlljTFan9OaP73\nvqSZZrYfUOPloQJVG6slPH/xtej1IODdHN8vOR6rdc6VqoIOGJIeltRL0nnR+w8knVToTsxs76gT\n7cXAT6rEai8gHIzeIdzPeCSK0PYEbjSz1cCNpKT5oMdqnXOlqqADhpl1MbMXzWxe9L6Xmf0k3/cU\ndaWVtERSmaQWklpGr1dG6zwtqYuk/QnTsQ4H3gZmSmoB9AU+lTR3e4tMEo/VOudKVaGXpO4g3F9Y\nD6DQfvy0uhxIlRju+cCL0b7+CXQys73qcn9x8litc64UFZqS2lXSa1X+JbyhLgeirBgu4dLVN4C/\nmVl/wkN/ZcAnNW3DY7XOOVc8Fm6Q51nJ7C+Eew0PS+prZicDYyQdW6eDMSsHDgbWATcBXyEktA4E\nxua6LFUlVttv0qRJdTmkorv33nvZZZddcsZqfWL7ZPFakikttcRZx5AhQ2ZLOjjvioVEqQiXil4A\nVgMfAX8DOhby3W35oUoMV1siteVAi3zf91ht8qSlDslrSaq01FIKsdq8l6TMrAFwsKSvm1lTQqvx\nVdt+DCucmbUEViu0CxlLmA9jZTH3WVsLFy7kzDPPZMmSJTRo0IDvfOc7jBv35TmfFi9ezFlnncXG\njRvZtGkTp5xyisdqnXMlIe8BQ6Hx3wXAJEmVdbFTM7uQ0CvqbaAdIQl1ZdYq3YAnzawF8AXwq7rY\nbzE1atSI66+/nr59+7Jq1Sr69evH0KFD6d69+1br9erVizfeeCOmUTrn3PYr9Kb382Z2CfBHQudZ\nACR9tp37PZ8vd7FFUicIz20QbnB3JNzPeMbMHlGeXlJxatu2LW3btgWgefPmdOvWjY8++uhLBwzn\nnCtVhcZqzwG+B0wDZkc/s7ZnhwV2se0GzJC0WuHhvpeBkduzvziUl5fzxhtveFzWOZcqBZ1hSNq3\nrnaowrrYzgOuNrNWwBrCw3x5D1B1Havd1rgshKTDSSedxI033kiLFi3qbCzOORe3QmO1Z+ZaLum+\n7dppFJ/NHDDMbAJQoWhipWjZGMJZTQXhXscaST/Isa3ExGo3bNjAFVdcwSGHHMIpp5xSq215VDB5\nvJZkSkstpRCrLfQexiFZr3cBjgJeB7brgFEISXcBdwGY2f8Q5srItd7twO0AXbt21eDBg4s1pBpJ\n4qyzzuLwww/nxhtvrPX2pk6dSly11KW01AFeS1KlpZZSqKPQS1Lfz35vZrsB9xdlRFv20UbSUjPb\nh/DU92HF3F9tTZ48mfvvv5+dd96Z2267jVatWnHHHXcwfPjwuIfmnHN1YpunaI2sBjrXYr/NgVfN\nbLGZbQJ+ClxjZv8xsxZmtgvwvpmtJbQ1f03S57XYX9F99atfZfbs2axdu5Zly5bRtGlTOnXqFPew\nnHOuzhQ6Reuf2DITXgOgO/CGfV0rAAAWpklEQVRwLfa7jBCr7UiYUGmrJ9csNK1qK6nCzBoTekod\nKimxfcA9VuucS7tC72FMzHq9Afi3pJz3FPKpEqu9O9c60aPqFdHbxtFP/rvzCeGxWudcGhV6wBgu\n6bLsBWZ2TdVlhdDWXWl7EiZUmgt8TDjbeCvafkPC8x4HALdK+nu+bXus1jnniqfQWO3rkvpWWfam\npF7btdOtu9Juii49DQduktS5yrotgceA70ual2NbHqtNsLTUAV5LUqWlllKI1ebrHnseob14JfBm\n1s8C4A+FdDesZrvlVOlKm2f5eMLZR2K71W7atElnnHGGxo0bVyfb8w6cyeO1JFNaaimFbrX5UlL/\nDziecL/h+KyffpJO34YDWE7RXN8Wve5PuKH+qZm1js4sMLMmwNeBf9Z2f8WUidXedttt7LLLLrRv\n356nn3467mE551ydqfGAIek/ksoljZL0b0KbDgHNoucjqmVmF5rZO2b2qJlNN7MvogaG2R4D1pvZ\nGuBm4LToaPdzYGm0/DOgXNJT21ljvfBYrXMu7QqN1R4P/IbQinwpIQ77DtCjhq/l7EgLW3WlvYyQ\nhrpP0qFZ3/0E+LGyWoUkncdqnXNpV+iDe78EDgXeVWhEeBTwSnUrF9iRFknTCGcQqeKxWudcGhUa\nq10v6VMza2BmDSS9ZGbXVLeyCutIm88FUdPDWcAPVcCT3h6rdc654ik0VvsC4ZLSr4FWhMtSh0j6\nag3fKSdPR9poeSfgKUk9s5btBSwn3C/5BeGp73Oq2Y/HahMsLXWA15JUaaml5GO12hJrbUq4fNUI\nOAu4EGiV5zvlZEVkgQnkiMYCnYB5NWynxs+zfzxWmzxpqUPyWpIqLbWkIVabOahUAh2AwZLuBe4k\nPHRXFGbWNuvtSMKESonmsVrnXNoVdMAws28DjwC/ixa1Bx4v8Lt7m9ki4GJCG5BFZvajKHJbTmj/\n0cPM1pvZ/Ohr15rZP8zsTcKZSauCK4qJx2qdc2lX6E3v7wH9gb8DSHrPzNrU9AVF0dlIWfZnZvZP\nQuT2c+BV4CuSPsxsU9IZ0XoXE1qIJP7uscdqnXNpV2is9gtJmy9BmVkjtrN7bJXI7feAyZI+BJC0\nNGu9MmAE4fJXSfFYrXMujQo9w3jZzH4MNDGzoYSH8v60PTvU1t1qfwI0NrOphEmVbtKWecJvBC6N\nlhfEY7XOOVc8hcZqGwBjgGGAAc8Cd6qQL+feXjnhUtOE6M+jgCbAdMJZRRdCS/XzzWwwOSZZytqW\nx2oTLC11gNeSVGmppeRjtcA+hUSttvWHKHILXA5MyFp+F/BN4FfAomi9JYQpYfN2x/VYbfKkpQ7J\na0mqtNSShljt5iSUmT26TYeswjwBHGlmjcxsV2AA8I6kKySVKdw4Pw2YojrojltMHqt1zqVdvgOG\nZb3erw7325yQjnqbMD94BeHJ7qcVTZJkZuVm9g/CTe8j6nDfReGxWudc2uU7YKia17W1DDgaOBzo\nLmkX4CRgUJX1hkg6QFLLOtx3UbRt25a+fcOkhNmxWuecS4t8B4zeZrbSzFYBvaLXK81slZmt3J4d\nVonVDtCWpoIzqPK8RqnyWK1zLo1qjNVKaljXO1T1nWzHAH/JXhV4zswE/E7S7fm27bFa55wrnoJi\ntXW+0y93sh0C/BY4QtKn0bJ2kj6Onv5+Hvi+wvwZVbflsdoES0sd4LUkVVpqKflYbbF+yOpkC/QC\n3ge61LD+BHJ0uq3647Ha5ElLHZLXklRpqSUNsdqiiuYFnwycIendrOVNzax55jXhgcFEd6z1WK1z\nLu3iOmBkYrX/Jsx38aKZVZrZW9HnhwJLzWwNIW7bFzgwjoEWymO1zrm0i+uAkR2rbS2pCXAy4XkM\nJL0oqUm0vBmwAngsprEWxGO1zrm0q/cDxnbEao8C3pf073oaYq15rNY5l0aFdqutMyo8VptxGvBg\nIdv2WK1zzhVPYmO10fKdgI+BHpI+qWZbHqtNsLTUAV5LUqWlFo/V1jJWC5wAPFfodj1WmzxpqUPy\nWpIqLbV4rDaP6mK1WUZR4OWoYjnnnHNo06YNPXv2rHG9V155hfvvv58pU6bQp08f+vTp47Fa51yq\n1Ps9jEgmVrsvoSPui2YGsAvQStJnZnYJITnV28x2k3RjHAMdPXo0F1xwAWeeeWaN6x1xxBGZsyLn\nnEulWGO1khpLaqQQnz0FeCk6WPQERgMtgIOA48yscxwDHThwIHvssUccu3bOuUSJNVZrZj/I+ij7\n8lM3YIak1ZI2AC8DI+t3pM4557IlIlYbzbZ3DHBBtNo84GozawWsAYYDs/Jtu9BY7fbEZZ1zbkcX\n1z2Mqo4HXpH0GYCkd8zsGkKX2gpgLrAh1xerxGqZdEzTvDubOnXqNg1uyZIlVFZWbvP3aqOioqJe\n91csaakDvJakSkstJVFHIVGquv4hK1YbvX8M+O8a1v8f4Px82y1WrHbBggXq0aNHUbZdHY8KJo/X\nkkxpqcVjtQUws90IU7M+UWV5m+jPfYBvEFO8dtSoURx22GHMnz+fsrIy7rrrrjiG4ZxzsUvCJamR\nhIfzKqssfzS6h7Ee+J629JyqV02aNGHjxo107dqVefMS3WHdOeeKKq4zjN8AfzWzz4GLgQPNbJaZ\nHZG1zp1AY2BXYpzre/To0TzzzDNx7d455xIjrjOM84FjCc9jVEqSmfUCJhEOHnsA44GDCXN7zzaz\nJ+M4yxg4cCDl5eX1vVvnnEucej9gVGlvfrekG6KPmhIODhDmynheUWrKzJ4nxG5rvI/hsVrnnCue\n2J/DMLORwK+ANkDmb/L2wMKsry2Kln2Jx2qTLS11gNeSVGmppRTqiP2mt6THgMfMbCDwC+DrhP5S\nX1q1mu/fDtwO0LVrVw0ePLjOx1heXk7Tpk0pxrarM3Xq1HrdX7GkpQ7wWpIqLbWUQh2xx2ozJE0D\n9jezPQlnFB2yPi4jzIvhnHMuJnG3Nz/Aoja1ZtYX2An4FHgWGGZmu5vZ7sCwaFm922+//dh///15\n6623/DkM59wOLe725nsBzcxsHbAJeDB66vAzM5tIuI+xE7Ac6ApMr++B/v73v6dZs2aceeaZ/hyG\nc26HFmt7c8KMen+R1ERSU0ljs9bpD1wkaSegE/BO/Q/T25s751xG7LHaatZpAQwkzImBpHXAunzb\n9litc84VjymGWeLMrJzwUF5P4FHCTe6PgUskvWVmfQjJp7eB3sBsYFyO9iFVY7X9Jk2aVOfjXbJk\nCVdccQX33HNPnW+7Oj6xffJ4LcmUllrirGPIkCGzJR2cd8VCOhTW9Q9Rt1rCjHrNomXDgfei1wcT\n2pkPiN7fBPwi33a9W23ypKUOyWtJqrTU4t1q85C0UlJF9PppoHFWrHaRpL9Hqz4C9I1pmM4554g/\nVrt3Vqy2fzSeTyUtARaaWddo1aMIl6fqncdqnXMuKNpNbzO7EDiP8Bd9O8IZwpWSJmat9hhwiJmt\nJ8yqd5okmdk3o+/808z+BbwFnF2ssdbEY7XOORcUMyWV6UhbCXQETsx8IKkTgJldRpiC9T5Jh2Z9\ndx7hnsbvCDfC887nXSzerdY554KiHDBydaQ1sy9lWSVNM7NOOZa/E21nm/brsVrnnCueohwwVKUj\nbTH2keHdapMtLXWA15JUaamlFOqIvVttbcm71SZaWuoAryWp0lJLKdSRmG61zjnnks0PGHl4rNY5\n54KiX5Iys72BWYSnujeZ2UWEJ7fPAZoQOtfuEUVrP5DUNZqF73agFTDDzD6Q1KXYY83FY7XOORcU\n7QxDUidJyyUtkVQmqYWklpLKgDGE2Gwf4BOgo6TGwJHR158EVgIHALsCa8yse7HGWhPvVuucc0Hc\n3WofAiZL+hBA0tJotf7AvyR9EH3nIUIr9Bqf9vZYrXPOFU/c3Wp/AjQGehAuTd0k6T4zOxk4RtH8\nGGZ2BqER4QU5tuXdahMsLXWA15JUaamlFLrVxh2rbQT0I/SKagJMN7MZQK4n9nIe2TxWm2xpqQO8\nlqRKSy2lUEfcB4xFwHKFeS4qzWwaYf6LRUCHrPXKCPNlOOeci0ncsdongCPNrJGZ7QoMIEzFOhPo\nbGb7mtlOwGmEex71zmO1zjkXxHWG0Rx4lXBgeJfQgBDCHBjzAKIute8BG4GfS3orjoF6rNY554K4\nzjCWAUcD3wMGAV0k7QJ8NWudXxLSUu9Jurr+hxh4rNY554Kkxmqr7WRbE4/VOudc8SQyVpu1Xifg\nKUk9a9iWx2oTLC11gNeSVGmppRRitXkn/S7GD1AO7An8LzADaBq9f49weSqzXidgXqHb7dKlS56p\nzrfPggUL1KNHj6Jsuzo+sX3yeC3JlJZa4qwDmKUC/o5Naqz23XiH5ZxzrqqkxmoTw2O1zjkXJDJW\na2ZdCZeqmgGNzGwT8EdJo+p7oB6rdc65IK4DxjLgWOBzwoGji6QPzawNgKT5wO4AZtYQ+Ai4PI6B\nDhw4kPLy8jh27ZxziZLYWG2Wo4D3Jf0737Y9Vuucc8WT6FhttO7dwOuS/reabXmsNsHSUgd4LUmV\nllo8Vlv7WO1OwHJgr0K267Ha5ElLHZLXklRpqcVjtfnli9UeSzi7+CSuATrnnAuSHqsdBTwYy8gi\nHqt1zrmgqGcYZnYhcB5hatV2QF/gysznkt4xswVAZbToOW3pVns/8E2gu5mNA0ZLmlPM8ebisVrn\nnAuKfUnqfMJlpUqgI3AigKROsDkyOwToRrg8NdPMukt6m9DW/FRJjxR5jDXyWK1zzgVFO2BUic/e\nLekGM6uaZ+0P/EvSB9F3HgJOIJyRbDOP1TrnXPEU7YAh6btmdgwwRNLyalZrDyzMer+IcB8j42oz\n+xnwInC5pC+qbqBKrJZJxzTNO7apU6cWVEPGkiVLqKys3Obv1UZFRUW97q9Y0lIHeC1JlZZaSqGO\nuFNSlmNZ5sGQK4AlhGjt7cBlwM+/tLJ0e/Q5Xbt2VTEmUS8vL6dp06b1OkF7KUwIX4i01AFeS1Kl\npZZSqCPulNQioEPW+zLgYwBJi6OI8BfAPYTLV84552IS9wFjJtDZzPY1s52A0wj3PDCzttGfRrhZ\nHktEyWO1zjkX1MslKTPbG5gFtAA2mdlFQHdJK83sAuBZoCHhGYxHovUbRykqgGnASfUx1qo8Vuuc\nc0FRDxiZ+GykrJp1ngaeBjCzfxJiuMuASkkys17AJEkVub5fbB6rdc65IO6b3pvliuFGHzVly43w\nGnms1jnniieWbrXVyXSxlbTczEYCvwLaACMkTa/mO96tNsHSUgd4LUmVllq8W+12drGtsmwg8EIh\n3/dutcmTljokryWp0lJLKXSrjTsllZekacD+ZrZn3GNxzrkdWSIPGGZ2QBSnxcz6Eh7e+zSOsXis\n1jnngmL2ksrZqVbSxOjzDsB9wN7AJqKntSMnAReZ2e7RZ49Fp031zmO1zjkXFDMllbNTbZYNwA8l\nvW5mzYHZwHCFG96vAW8RbnZ/YWZtijjOGnms1jnngqIcMArpVCtpMbA4er3KzN4hNCN8m3Bm8mtF\nzQYlLS1kvx6rdc654ilarDY7Ihu9nwBUZC5JVVm3E+Fp7p4KT3/PIczGdwywFrhE0sxq9uOx2gRL\nSx3gtSRVWmophVht7A/umVkz4FHgIkkro8WNgN2BQ4FDgElmtl+u+xjybrWJlpY6wGtJqrTUUgp1\nxJqSMrPGhIPFA5ImZ320CJgcRYRfI9z49litc87FKLYDRhSbvQt4R9Jvqnz8OPC1aL0uhFhtdZMw\nFdWoUaM47LDDmD9/vsdqnXM7tKJfkqquUy3QCzgD+Ed0zwLgxwrNCO8G7jazecA64Ky4YrUPPvhg\nHLt1zrnEKeYUrZ2y3ubqVPs3cs+4h6R1wOlFGJZzzrntlMgnvZ1zziVPorrV1paZrQLmxz2OOrIn\nMd23qWNpqQO8lqRKSy1x1tFRUut8K8Ueq61j8wvJEpcCM5uVhlrSUgd4LUmVllpKoQ6/JOWcc64g\nfsBwzjlXkLQdMG7Pv0rJSEstaakDvJakSkstia8jVTe9nXPOFU/azjCcc84ViR8wnHPOFSQVBwwz\nO8bM5pvZv8zs8rjHUxtmVm5m/zCzOWY2K+7xbAszu9vMlkYtXTLL9jCz583svejP3eMcY6GqqWWC\nmX0U/W7mmNnwOMdYCDPrYGYvmdk7ZvaWmY2Llpfc76WGWkrx97KLmb1mZnOjWq6Klu9rZn+Pfi9/\nNLOd4h5rtpK/h2FmDYF3gaGELrczgVGS3o51YNup6jwipcTMBgIVwH2SekbLrgU+k/Tr6GC+u6TL\n4hxnIaqpZQLVzOmSVGbWFmhbZWbLE4HRlNjvpYZaTqH0fi8GNJVUEXXt/hswDriY0Kn7oWgiurmS\n/i/OsWZLwxlGf+Bfkj6IelA9BJwQ85h2SJKmAZ9VWXwCcG/0+l6+PFVvIlVTS8mRtFjS69HrVUBm\nZsuS+73UUEvJiaZuqIjeNo5+ROjS/Ui0PHG/lzQcMNoDC7PeL6JE/0cUEfCcmc2OZhMsdXtF0/Fm\npuWNbX72OnKBmb0ZXbJK/GWcbNHMll8B/k6J/16q1AIl+Hsxs4ZRp+6lwPPA+8AKSRuiVRL3d1ka\nDhi5Ot6W8nW2wyX1BY4FvhddGnHJ8H/A/kAfwnz018c7nMJVM7NlScpRS0n+XiRtlNSH0M27P9At\n12r1O6qapeGAsQjokPW+DPg4prHUmqSPoz+XAo8R/odUyj6Jrj1nrkEvjXk8203SJ9H/yTcBd1Ai\nv5tqZrYsyd9LrlpK9feSIWkFMJUwJXVLM8v0+Evc32VpOGDMBDpH6YKdgNOAJ2Me03Yxs6bRzTzM\nrCkwDJhX87cS70ngrOj1WcATMY6lVjJ/wUZGUgK/mxpmtiy530t1tZTo76W1mbWMXjcBvk64J/MS\ncHK0WuJ+LyWfkgKIYnQ3Ag2BuyVdHfOQtouZ7Uc4q4DQSfj/lVItZvYgMJjQpvkTYDxhut1JwD7A\nh8A3JSX+ZnI1tQwmXPYQUA6cm7kPkFRmdgTwV+AfwKZo8Y8J1/5L6vdSQy2jKL3fSy/CTe2GhH+4\nT5L08+jvgIeAPYA3gNMlfRHfSLeWigOGc8654kvDJSnnnHP1wA8YzjnnCuIHDOeccwXxA4ZzzrmC\n+AHDOedcQRrlX8W5HZuZbSREOTNOlFQe03Cci43Hap3Lw8wqJDWrx/01yuon5Fxi+CUp52rJzNqa\n2bRoLoZ5ZnZktPwYM3s9mvPgxWjZHmb2eNQob0b0AFdmTofbzew54L6oMd11ZjYzWvfcGEt0DvBL\nUs4VoknUVRRggaSRVT7/b+BZSVdH87PsamatCX2NBkpaYGZ7ROteBbwh6UQz+xpwH+EpZYB+wBGS\n1kSdiv8j6RAz2xl4xcyek7SgmIU6VxM/YDiX35qoq2h1ZgJ3R43xHpc0x8wGA9Myf8Fntd04Ajgp\nWjbFzFqZ2W7RZ09KWhO9Hgb0MrNMX6HdgM6AHzBcbPyA4VwtSZoWtaEfAdxvZtcBK8jdmrqmdvyV\nVdb7vqRn63SwztWC38NwrpbMrCOwVNIdhG6qfYHpwCAz2zdaJ3NJahrwrWjZYGB5NfNTPAucF521\nYGZdog7GzsXGzzCcq73BwI/MbD1hHvAzJS2L7kNMNrMGhPkmhgITgHvM7E1gNVtajFd1J9AJeD1q\n672MhE3X6XY8Hqt1zjlXEL8k5ZxzriB+wHDOOVcQP2A455wriB8wnHPOFcQPGM455wriBwznnHMF\n8QOGc865gvx/D3KBs/prQHYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x113ceff50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot feature importance using built-in function\n",
    "from xgboost import plot_importance\n",
    "plot_importance(bst)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
